DOCUMENT RESUME 



ED 452 111 



HE 033 996 



PUB DATE 
NOTE 



AUTHOR 

TITLE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Hu, Shouping; Kuh, George D. 

A Multilevel Analysis on Student Learning in Colleges and 
Universities . 

2000 - 11-00 

5 Op . ; Paper presented at the Annual Meeting of the 
Association for the Study of Higher Education (25th, 
Sacramento, CA, November 16-19, 2000) . 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

MF01/PC02 Plus Postage. 

Educational Experience; Higher Education; *Institutional 
Characteristics; *Learning; Models; Productivity; 
♦Undergraduate Students 

Hierarchical Linear Modeling; ^Multilevel Analysis 



ABSTRACT 



This study tested a learning productivity model for 



undergraduates at four-year colleges and universities using hierarchical 
linear modeling. Student level data were from 44,328 full-time enrolled 
undergraduates from 120 four-year colleges and universities who completed the 
College Student Experiences Questionnaire between 1990 and 1997. 

Institutional level data came from various sources. Institutional 
characteristics had effects on student engagement and gains from colleges and 
universities, and the institutional effects on gains were through effects on 
engagement, effects on gains, or effects on the conversion rates from effort 
to gains. The effects also varied depending on the types of gains students 
had from colleges and universities. The findings have implications for the 
conversations on collegiate quality and research on college impacts on 
-students. Three appendixes contain instrument scale items and results from 
the analyses. (Contains 1 figure, 4 tables, and 37 references.) (Author/SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



ED 452 111 



A multilevel analysis on student learning in colleges and universities 






PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
been.granted by 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Shouping Hu, Assistant Professor 
Department of Educational Administration and Supervision 
College of Education and Human Services 



Seton Hall University 
400 South Orange Avenue 
South Orange, NJ 07079 
Tel: 973-275-2324 
Fax: 973-761-7642 
Internet: hushoupi@shu.edu 



OHVr^f D PH PA P™,l NT 0F education 

Ottico ol Educational Research and Improvement 

educational resources information 

J CENTER (ERIC) 

or This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



George D. Kuh, Professor 
Center for Postsecondary Research and Planning 
School of Education 
Indiana University 
201 N. Rose Avenue 
Bloomington, IN 47405 
Tel: 812-856-8383 
Fax: 812-856-8394 
Internet: kuh@indiana.edu 



Prepared for presentation at the annual meeting of the Association for the Study of Higher 
Education (ASHE), Sacramento, CA, November 2000 




BEST C0PV AVAILABLE 



2 



Student Learning in American Colleges and Universities: 
Unraveling the Effects of Institutional Characteristics 

Abstract 

This study tests a learning productivity model for undergraduates at four-year colleges 
and universities using hierarchical linear modeling. Student level data were from 44,238 full- 
time enrolled undergraduates from 120 four-year colleges and universities who completed the 
College Student Experiences Questionnaire between 1990 and 1997. Institutional level data came 
from various sources. Institutional characteristics had effects on student engagement and gains 
from colleges and universities, and the institutional effects on gains were through effects on 
engagement, effects on gains, or effects on the conversion rates from effort to gains. The effects 
also varied depending on the types of gains students had from colleges and universities. The 
findings have implications for the conversations on collegiate quality and research on college 
impacts on students. 



Student Learning in American Colleges and Universities: 
Unraveling the Effects of Institutional Characteristics 

Introduction 

Student learning is one of the central functions of undergraduate education. Despite 
considerable evidence that college attendance is associated with numerous desirable short and 
long-terms effects (Astin, 1993b; Pascarella & Terenzini, 1991), various groups from inside and 
outside the academy continue to press for improvement in the quality of the baccalaureate 
experience (Educational Commission of the States, 1995; Kellogg Commission, 1997; 
Wingspread Group on Higher Education, 1993). Such expectations are appropriate, given that 
more students than ever are participating in higher education and the knowledge, skills, and 
competencies acquired during college are considered essential for the post-college success of 
individuals, preparation of an informed citizenry, and continued expansion of an information- 
based economy. 

Unfortunately, there are few external incentives for institutions of higher education to 
improve undergraduate education. In part, this is because the conversation about “quality” tends 
to center on such things as institutional resources and reputations, variables that are only weakly 
linked to learning. They also overlook the key factors that can enhance student learning — the 
investments that institutions make to engage students in proven instructional practices. The 
weight of the evidence in the higher education literature points to student engagement as the key 
factor in student learning and personal development (Astin, 1993b; Pascarella & Terenzini, 
1991). That is, what matters most to learning is the quality of effort students expend on using the 
institution’s resources and facilities, such as the amount of time they spend studying or using the 
library. Therefore, one of the key challenges for all colleges and universities is to arrange their 
resources for learning so that students spend more of their time on the activities that matter to 



their education. We need to learn more about this untapped dimension of collegiate quality 
including the characteristics of institutions that are more or less effective in promoting student 
learning. 

The Carnegie Classification of Institutions of Higher Education (1994) was originally 
created to promote research on higher education issues and problems (Carnegie Foundation for 
the Advancement of Teaching, 1973) and is the most often used institutional characteristic in 
studies of the college student experience. The classification is primarily based on such variables 
as numbers and types of degree programs (including graduate and professional education), 
numbers of degrees granted, and research productivity. However, these measures are not strong 
predictors of many important outcomes of the undergraduate experience. The best predictor of 
what students gain from college (after controlling for student background characteristics) is 
student effort — the extent to which students use institutional resources to educational advantage 
(Astin, 1984, 1993b; Kuh, 1999; Pace, 1984; Pascarella & Terenzini, 1991). The constellation of 
factors that represent student effort is not considered in the Carnegie classification. Even so, the 
institutional types popularized by the Carnegie classification system continue to be widely used 
in research and commentary on the undergraduate experience. For example, the Boyer 
Commission on Educating Undergraduates in the Research University (1998) and Sperber (2000) 
sharply criticized the sub-par quality of baccalaureate education at research universities. National 
college student databases such as The National Survey of Student Engagement, the College 
Student Experiences Questionnaire, and the Cooperative Institutional Research Program 
typically report findings using the Carnegie institutional types. 

After reviewing the 1994 classification system, The Carnegie Foundation for the 
Advancement of Teaching is revising its schema to make it more flexible and comprehensive for 




4 5 



both research and policy analysis purposes (2000). Following fairly modest changes released this 
fall, fundamental changes to the 1994 schema are promised by 2005, perhaps with the addition of 
some features that more clearly distinguish among institutions that emphasize undergraduate 
education. 

Purpose 

This study is based on the proposition that colleges and universities can promote student 
learning in multiple ways. To examine this proposition we will delineate and test a learning 
productivity model for colleges and universities. In the tradition of educational production 
function research (Bowles, 1970; Coleman, et al., 1966; Hanushek, 1979) and college impact 
studies (Astin, 1993a; Ethington, 1997; Pascarella & Terenzini, 1991), we intend to determine 
the contributions of a variety of institutional characteristics to a range of important 
undergraduate learning outcomes, including self-directed learning, analysis and synthesis skills, 
writing ability, team work, and so forth. 

The main research question guiding the study is: what institutional characteristics 
contribute to undergraduate learning from colleges and universities at institutional level? 
Institutional learning productivity is defined in this study as the average amount of effort 
students expend in educationally purposeful activities (those empirically linked to desired 
outcomes of college), the average gains reported by students in 23 important areas of learning 
and personal development, and the conversion rate of student effort to student gains. 

Perspectives 

Studies on Educational Production Function 

Research on learning productivity dates back to the publication of Equality of 
Educational Opportunity, more commonly known as the “Coleman Report” (Coleman, et al., 
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1966). In this study the researchers estimated an input-output model to determine the relationship 
between student achievement and inputs, which was defined as student and family characteristics 
and school resources. Subsequently, economists introduced the concept of educational 
production function to describe the relationships between observed student outcomes and inputs 
(Hanushek, 1979). The early educational production function model was typically estimated 
using multiple regressions, where the outcome variables were regressed on all input variables. 

Although there are different views about the degree to which school inputs affect student 
performance (Butless, 1996), most economists and educational researchers acknowledge that 
student achievement is influenced by a set of factors including student characteristics and 
characteristics of the learning environment (broadly defined) in which learning occurs. Inputs 
refer to the personal qualities the student brings to college, such as ability, attitudes, aspirations, 
and socio-economic status. Outcomes refer to student “talents” to be developed, such as 
intellectual functioning and interpersonal competence. The environment refers to the array of 
student experiences during college including interactions with faculty members and peers as well 
as structural and perceptual features of the institution (Astin, 1993a, 1993b; Pace, 1990; 
Pascarella & Terenzini, 1991). 

Two models are frequently used to explain undergraduate learning and personal 
development in college. In Astin’s (1993a) Input-Environment-Output (I-E-O) model, student 
learning (outcomes) are presumed to be a function of the interactions of inputs and the 
environment which encompasses student perceptions and behavior as well as an institution’s 
human, financial, and physical resources. In Pascarella’s (1985) model student learning is 
presumed to be a function of the interaction of student background characteristics (inputs), 
institutional characteristics (size, affluence, student-faculty ratio), student interactions with 
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agents of socialization (faculty, peers), student perceptions of the environment, and student 
quality of effort. The preferred methodological strategy is sequential multiple regression, where 
the input variables are treated as control variables and environment variables are the independent 
variables to be regressed on outcome variables. 

The Learning Productivity Model 

The conceptual approach used in this study splits the environment component of the 
Astin I-E-0 model into two sets of variables, which makes it more consistent with the Pascarella 
model. One set of variables is “quality of effort” (Pace, 1984), which is defined as the frequency 
of student engagement in various educationally purposeful college activities that are empirically 
linked with desired outcomes of college. The other environment variable includes such measures 
as institutional type, selectivity, and student perceptions of the campus climate including (a) the 
degree to which the school emphasizes scholarship or critical analysis and (b) the quality of 
relations between students and faculty, students and administrators, and students and students. 
This approach allows us to better understand how learning is influenced by what students do, 
what institutions provide, and what students think about their institutions. 

This learning productivity model proposes that institutions affect student learning in at 
least three ways as illustrated in Figure 1. First, a college or university school can arrange its 
intellectual resources and design policies and practices to engage students at higher average 
levels in educational purposeful activities. Examples of some of these successful approaches are 
learning communities, small classes for first-year students, capstone courses for seniors, and 
intrusive developmental advising. Second, student learning can be improved by increasing the 
conversion rate of student effort to gains. That is, at high performing institutions the yield of 
student effort in terms of gains may be greater compared with other colleges and universities. A 
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third approach is to increase the average “net” amount that students gain from their educational 
experience, independent of their engagement in college activities or the effort to gain conversion 
rate. In Figure 1 this is illustrated as the gross effect of institutions on student gains from college 
which is a function of these three components: (a) the effects of institution-level effort, (b) the 
effects of the effort-gain conversion rate, and (c) the net effects on institutional average gain. If 
we can better understand the effects of these three sources of institutional contributions on 
student learning and coordinate their influence so they have complementary, positive effects on 
students, we may be able to increase learning productivity and improve the quality of 
undergraduate education. 

(Insert Figure 1 About Here) 

Methods 

Data Source and Instrument 

The sample is composed of 44,238 full-time enrolled undergraduates at 120 four-year 
colleges and universities who completed the third edition of the College Student Experiences 
Questionnaire (CSEQ) between 1990 and 1997. No transfer students are included in the sample. 
That is, all the students started college and attended only the institution at which they were 
enrolled when they filled out the CSEQ. Using only native, full-time students was done because 
the purpose of the study was to examine the impact of institutional characteristics on students. 
Including part-time and transfer students would have confounded the findings, making the results 
very difficult to interpret. The institutions included 20 research universities (RUs), 14 doctoral 
universities (DUs), 41 comprehensive colleges and universities (CCUs), 15 selective liberal arts 
colleges (SLAs), and 30 general liberal arts colleges (GLAs) (Carnegie Foundation for the 
Advancement of Teaching, 1994). Table 1 shows the background characteristics of the sample. 
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Women (61%) and first-year students (41%) were somewhat over-represented compared with the 
national profile of undergraduates attending four-year colleges and universities before 1998. 

Also, about half of the students (44%) were majoring in a pre-professional area. 

(Insert Table 1 about here) 

The College Student Experiences Questionnaire (CSEQ) is well suited to address the 
purposes of this study in that it is based on a simple but powerful premise related to student 
learning. That is, the more students put into using the resources and opportunities an institution 
provides for their learning and development, the more they benefit (Kuh, Vesper, Connolly, & 
Pace, 1997). Also, the CSEQ has excellent psychometric properties and high to moderate 
potential for assessing student behavior associated with college outcomes (Ewell & Jones, 1996). 
The items are well constructed and responding to the questionnaire requires that students reflect 
on what they are putting into and getting out of their college experience. As with all survey 
questionnaires, the CSEQ relies on self-reports from students. Examinations of the validity of 
self-reports (Baird, 1976; Lowman & Williams, 1987; Pace, 1984; Pike, 1989, 1995; Pohlman & 
Beggs, 1974; Turner & Martin, 1984) indicate that they are generally valid if they meet three 
conditions: (1) when the information requested is known to respondents, (2) if the questions are 
phrased clearly and unambiguously, and (3) if respondents think the questions merit a serious 
and thoughtful response (Pace, 1985). CSEQ items satisfy all these conditions. The distributions 
of responses on the Activities and Gains scales are approximately normal and the psychometric 
properties of the instrument indicate it is reliable (Kuh, Vesper, Connolly, & Pace, 1997). The 
Gains items ask students how much they think their college or university experience contributed 
to their own growth and development and Gain scores are generally consistent with other 
evidence, such as results from achievement tests (Pace, 1985; Pike, 1995). For example, Pike 
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(1995) found that student reports of their experiences using the CSEQ were positively correlated 
with relevant achievement test scores. In this sense the progress students report is a 
“value-added” judgment (Pace, 1990b). 

Variable Specifications 

Seven dependent variables were specified for this study. The first is EFFORTSUM, a 
measure of the quality of student effort. The second is GAINSUM, students’ self-report of the 
progress they have made toward desirable outcomes of college. The original measurement of 
these variables is presented in Table 1. They are defined for this study as follows: 

• EFFORTSUM (the sum of student responses to the 14 CSEQ quality of effort scales covering 
128 items which ask students how often they engaged in certain college activities. Response 
options for the effort items are 1= “never,” 2= “occasionally,” 3= “often,” and 4= “very 
often.”). 

• GAINSUM (the sum of student responses to the 23 CSEQ gains items which ask students 
how much their college or university experience contributed to their growth and development 
during college. Response options for the gains items are 1= “very little,” 2= “some,” 3= 

“quite a bit,” and 4= “very much”). 

• Five measures of learning outcomes distilled from a factor analysis of the 23 Estimate of 
Gains items. They are Intellectual Skills, General Education, Personal/Social Development, 
Science/Technology, and Practical/Vocational Preparation (Kuh, et al, 1997). Pace (1990b) 
recommended using the Estimate of Gains factors when the number of respondents is large 
and the sample is from multiple institutions, as is the case in this study. The measures of the 
five gain factors were the sums of the response to the gain items clustered within each gain 
factor. The response options for the gains items are 1= “very little,” 2= “some,” 3= “quite a 
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bit,” and 4= “very much.” The gain items included in each gain factor are presented in 
Appendix A. 

The student-level control variables are defined as follows: 

• Sex (0=female, l=male); 

• Race or ethnicity was coded as a set of dummy variables: Asian Americans, African 
Americans, Latinos, Whites, and students from “other” backgrounds (American Indians and 
others), with Whites as the omitted reference group; 

• SES (the sum of parent education where l=neither parent a college graduate, 2=one parent a 
college graduate, and 3=both parents college graduates and the amount parents contribute to 
college costs where l=none to 4=all or nearly all). 

• Academic capital (the sum of grades where 5=A and 1=C, C- or lower and educational 
aspirations where 2=expect to pursue an advanced degree after college and l=does not expect 
to pursue an advanced degree). 

• Major fields (four major field clusters coded as dummy variables: humanities— arts, literature, 
history, philosophy, religion, foreign language; science and mathematics including computer 
science; social sciences— economics, political science, psychology, sociology; and 
professional/applied— agriculture, business, education, engineering, health-related fields such 
as nursing. Students who were undecided or indicated some “other” major were excluded 
from this analysis. Pre-professional was omitted as the reference group; 

• Year in college (first-year, sophomore, junior, and senior, with first-year omitted as reference 
group); 

• Hours per week devoted to attending and preparing for class (l=less than 20, 2=about 20, 
3=about 30, 4=about 40, and 5=about 50 or more). 
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The institution-level variables are defined as follows: 



• Institutional control (0=private, l=public); 

• Institutional selectivity (6=most competitive, 5=highly competitive, 4=very competitive, 
3=competitive, 2=less competitive, and l=not competitive from year 1996 Barron ’s Profiles 
of American Colleges). 

• Institutional type (RU, DU, CCU, SLA, GLA with CCU omitted as reference group); 

• Institutional environment measures: Eight environment scales that assess students’ 
perceptions of aspects of the college environment were clustered into three institutional 
environment measures to reduce the risk of multi-colinearity among variables and reduce the 
dimensions of institutional environment measures (Kuh, et al., 1997) (Appendix A). They 
represent the degree to which students perceive their institution emphasizes scholarly and 
intellectual activities, vocational preparation or practical issues in courses, and students’ 
perceptions of the quality of relations that exists among different groups (student-student, 
student-faculty, student-administration). 

Statistical Model and Data Analysis 

To test the learning productivity model, we used Hierarchical Linear Modeling (HLM) 
(Bryk & Raudenbush, 1992; Ethington, 1997, 2000), an approach especially well suited to 
examining questions that require data about student experiences and institutional characteristics. 
All student-level continuous variables were standardized as z-scores (M=0, SD=1), centered on 
the grand-mean of the sample of students. Institution-level continuous variables were also 
standardized as z-scores (M=0, SD=1), centered on the grand-mean of the sample of institutions. 
Because the variables are either dummy-coded categorical variables or z-scored continuous 
variables, the results are presented in the metric of student-level standard deviation units of 
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EFFORTSUM and GAINSUM (Lee & Smith, 1999). 

The estimates of learning productivity were done in two steps. First, we estimated a two- 
level model on the effects of institutional characteristics on EFFORTSUM (engagement in 
educationally purposeful activities). In this model, institutional characteristics were assumed to 
have a direct effect on the average value of EFFORTSUM after controlling for individual student 
characteristics. The student-level model was estimated by: 

(1) EFFORTSUM y = fi j0 + Pj\X tJX + ft jl X jj2 + ... + fi jp X ^ + £ IJ where i represented the /th 

student, j represented the y'th institution, and p represented the pth student level covariate. X 
represents student characteristics such as gender, race or ethnicity, academic preparation, and so 
on, and the coefficients of X represent how student characteristics affect EFFORTSUM. 

The institution-level model was estimated by: 

(2) PjQ =y ok +Vu Z j\ + r 2 k z j2 + - + r q k Z Jq + v jk where; represented they'th institution and^ 

represented the qth institutional level covariate. Z represents institutional characteristics such as 
institutional type, selectivity, environment, and so forth, and the coefficients of Z represent how 
institutional characteristics affect student effort. 

The second step was estimating a two-level model in which the sum of student-reported 
gains (GAINSUM) was the dependent variable and EFFORTSUM was the independent variable. 
This step serves two purposes. First, it determines if there are differences between institutional 
characteristics in the conversion rates of EFFORTSUM to GAINSUM. Second, it allows us to 
determine if institutional characteristics have a net effect on GAINSUM after taking into account 
the difference in EFFORTSUM and the difference in the conversion rates of EFFORTSUM to 
GAINSUM. In this model, institutional characteristics were assumed to affect both the average 
values of GAINSUM and the EFFORTSUM to GAINSUM conversion rate (the slope of 



EFFORTSUM) at the institutional level, when individual student characteristics are held 
constant. 

The student-level model was calculated as: 

(3) GAINSUM y = /3jQ + fi J{ EFFORTSUM y + f3 j2 X ij2 + ... + fi' Jp X ijp + e , where i,j,p, mdX 

had the same representation as in equation (1). (3 represents how student characteristics affect 
student GAINSUM. 

The institution-level model was estimated by: 

(4) ^0 =r'ok +r'u Zji+r'ikZji + - + r< l kZ jtl +v' jk and 

(5) =r'ok +r"\ k Zji + y 2 k Z J2 + ■■■ + Y q k Zj,, + V jk , where j, q and Z had the same representations 

as in equation (2). y and y represent how institutional characteristics affect institution-level 
average EFFORTSUM and the conversion rate of EFFORTSUM to GAINSUM. 

In equation (2), institutions are assumed to have direct effects on EFFORTSUM. In 
equation (4), institutional characteristics are assumed to have a direct (net) effect on student 
gains (GAINSUM), after controlling for EFFORTSUM, and in equation (5), institutional 
characteristics are assumed to have a direct effect on EFFORTSUM-GAINSUM conversion rate. 
To paint a more complete picture of institutional effects on student GAINSUM from college, we 
also need to understand the gross effect of institutions on GAINSUM (Pascarella & Terenzini, 
1991). The gross effect of institutions on student GAINSUM is defined as the combination of the 
direct effect of institutions on EFFORTSUM, the net effect of institutions on the EFFORTSUM- 
GAINSUM conversion rate, and the net effect of institutions on GAINSUM. We estimated the 
gross effect of institutions on GAINSUM by redoing the above analysis from equations (3) to (5) 
while excluding EFFORTSUM from the independent variables tested in the model (Pascarella & 



Terenzini, 1991). 

To examine how institutional type and environmental measures affected the five gain 
factors, the gain factors were introduced separately as dependent variables using the same 
analytical model as GAINSUM was the dependent variables. 

In the individual student-level model (level 1), we controlled for such student background 
characteristics as gender, race and ethnicity, major field, and class level, student SES, time spent 
on schoolwork per week, and educational aspirations. When GAINSUM was the dependent 
variable, EFFORTSUM was treated as the independent variable and was grand-mean centered. 

As mentioned earlier, all the student-level variables were centered around the grand mean for the 
sample, which allowed us to interpret the intercept as the mean outcome for each institution, 
adjusted for student characteristics in each institution (Bryk & Raudenbush, 1992). 

In the institution-level model (level 2), two sets of variables were analyzed. The first set 
was the five types of four-year colleges and universities — RUs, DUs, CCUs, SLAs, and GLAs 
(The Carnegie Foundation for the Advancement of Teaching, 1994). The second set of variables 
was composed of three aggregate measures of the environment mentioned earlier: scholarly and 
intellectual emphasis, vocational and practical emphasis, and quality of personal relations (Kuh, 
et al., 1997). In addition, institutional selectivity and institutional control were also included 
when estimating how well the two sets of institutional characteristics predicted learning 
productivity. 

Results 

Descriptive Statistics 

Table 2 presents descriptive statistics for both the student and institutional levels by 
institutional type. Because we used the grand mean-centered strategy for continuous variables, 
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anything above zero on mean values indicates an above-average score. Table 2 shows that 
EFFORTSUM and GAINSUM at SLAs and EFFORTSUM at GLAs were above average and 
below average at RUs, DUs, and CCUs. These results are consistent with those reported by Kuh 
and Hu (in press a). Students at SLAs reported the greatest gains in general education and 
intellectual development, GLA students the greatest gains in personal development and 
vocational preparation, and RU students the greatest gains in science and technology. 

In addition, students at RUs, DUs, and CCUs reported that their schools emphasized 
scholarly and intellectual activities less than did students at SLAs and GLAs. Students at RUs 
and DUs also reported lower levels overall of the quality of personal relations among various 
groups compared with students at the other three institutional types. Finally, RUs, DUs, and 
SLAs placed less emphasis on vocational and practical matters in courses compared with GLAs 
and CCUs. 

The Unconditional Hierarchical Models 

The reliability measures (lambda) in Table 3 indicate that the estimates of EFFORTSUM 
and GAINSUM in the model are reliable. To evaluate the effects of institutional characteristics 
we must partition the total variability in the dependent variable into its within-institution and 
between-institution components. When EFFORTSUM is the dependent variable, the within- 
institution variance was estimated as 0.917 and the between-institution variance as 0.085, 
resulting in an intraclass correlation of 8.5%. When GAINSUM was the dependent variable, the 
within-institution variance was estimated as 0.941 and the between-institution as 0.073. Thus, the 
intraclass correlation was 7.3%. These small intraclass correlations indicate that relatively little 
variance in either EFFORTSUM (8.5%) or GAINSUM (7.3%) is due to institutional 
characteristics (Table 3). The intraclass correlations when the five gain factors were dependent 



variables ranged from 3.3% (personal development) to 9.4% (general education), suggesting that 
relatively little variance in gains is a function of institutional characteristics. 

(Insert Table 3 About Here) 

The Within-Institution Models 

To test the within-institution (random coefficient) model, the dependent variables were 
regressed with student-level variables and the coefficients of level 1 variables were specified as 
random in the level 2 models. Because this study seeks to estimate the effect of institution-level 
variables on student learning, the influence of student-level variables is only briefly summarized. 
The results from the random coefficient model indicated that EFFORTSUM, GAINSUM, and 
the gain factors differed by student characteristics such as gender, race or ethnicity, class level, 
and major field, consistent with other studies using student as the unit of analysis in conventional 
regression analyses (Kuh & Hu, in press b). However, although most of the variance in 
EFFORTSUM (91.5%) was associated with student-level characteristics, these characteristics 
accounted for less than 7% for EFFORTSUM. This suggested that student EFFORTSUM rarely 
depends on who the students are but what students really do in college. Student background 
characteristics and EFFORTSUM explained about 37% for GAINSUM of the total variance at 
student level. The variance in gains associated with student characteristics and EFFORTSUM 
ranged from 12.3% (vocational preparation) to 32.0% (science and technology). The 
EFFORTSUM coefficient was 0.555 when GAINSUM was the dependent variable in the within- 
institution hierarchical model, suggesting that EFFORTSUM and GAINSUM were highly 
related, as one would expect from the previous research (Astin, 1993; Pace, 1990; Pascarella & 
Terenzini, 1991). The EFFORTSUM coefficients ranged from 0.276 (vocational preparation) to 
.474 (personal development) when the five gain factors were the dependent variables. 
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The Full Models 



The results from the full HLM are presented in Table 4. The two level models in the full 
HLM model were estimated simultaneously. Institutional sector, selectivity, institutional type, 
and institutional environment measures were included in the level 2 model, while all variables 
concerning student characteristics were included in the level 1 model. The gross effects models 
represent the effects of different institutional characteristics on student learning. The net effect 
models provide insight into how these effects are produced. 

(Insert Table 4 About Here) 

EFFORTSUM as Dependent Variable . When institutional sector, selectivity, institutional 
type, and the environment measures were examined at the institutional level, public colleges and 
universities had a significantly lower EFFORTSUM mean than private institutions. 
EFFORTSUM at RUs, and DUs did not differ from CCUs, but the SLA and GLA means were 
significantly higher than the CCU mean. Emphasizing scholarly and intellectual activities did not 
significantly contribute to EFFORTSUM, though high quality personal relations positively 
affected EFFORTSUM, while emphasizing vocational preparation and practical matters had a 
negative effect. 

GAINSUM as Dependent Variable . At the institutional level, selectivity had a negative 
net effect on GAINSUM. After EFFORTSUM was controlled, SLAs had lower GAINSUM 
mean than CCUs, but GAINSUM at other types of institutions did not differ significantly from 
CCUs. Institutions that emphasized scholarly and intellectual activities and practical and 
vocational activities had positive net effects on GAINSUM, while environments that emphasized 
personal relationships had a negative net effect on GAINSUM. At the same time, the 
EFFORTSUM to GAINSUM conversion rate (the slope of EFFORTSUM) did not vary 
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significantly with respect to institutional sector, selectivity, institutional type and institutional 
environment. 

The gross effects of institutional characteristics on student GAINSUM are reported in the 
bottom panel in Table 4. It is noteworthy that there were no significant net effects of institutional 
characteristics on the EFFORTSUM to GAINSUM conversion rate, which makes the different 
directions of institutional net effects on EFFORTSUM and GAINSUM worth examining more 
closely. Of particular importance is the comparison between SLAs and CCUs, where the a higher 
level net effect on EFFORTSUM and a lower level net effect on GAINSUM does not result in 
any differences in gross effect at SLAs and CCUs after controlling for the institutional 
environment measures. This suggests that the slight gross effect advantage of SLAs over CCUs 
on GAINSUM in the “institutional type model” was due to the institutional environments of 
SLAs, as indicated in Table 2. Also, there were no significant differences in the gross effect of 
environments that emphasized the quality of personal relations and vocational matters on 
GAINSUM. However, institutional environments that emphasized scholarly and intellectual 
activities had both net and gross positive effects on GAINSUM. 

General Education as Dependent Variable . After controlling for EFFORTSUM, 
institutional type had no net effect on general education gains at the institutional level. 

Institutions that emphasized scholarly and intellectual activities had positive net effects on 
general education gains, while environments that emphasized personal relationships had a slight 
negative net effect on general education gain. The EFFORTSUM to GAINSUM conversion rate 
(the slope of EFFORTSUM) was significantly higher for SLAs than in CCUs, suggesting that 
equal amounts of EFFORTSUM were converted to greater gains in general education for 
students at SLAs compared with their counterparts at CCUs. Institutions that emphasized 



scholarly and intellectual activities also had a slightly higher conversion rate from EFFORTSUM 
to general education gains. 

The gross effects of institutional characteristics on general education gains are reported in 
the bottom panel in Table 4. Here again, institutional environments emphasizing scholarly and 
intellectual activities had significant gross positive effects on general education gains. Also, 
students at SLAs and GLAs had a slight advantage in general education gains compared with 
students attending CCUs. 

Personal Development as Dependent Variable . At the institutional level, SLAs had a 
slightly smaller net effect on personal development gains (compared with CCUs), after taking 
into account EFFORTSUM. Institutions that emphasized practical and vocational matters had 
positive net effects on personal development. Neither institutional type nor environment affected 
the EFFORTSUM to GAINSUM conversion rate. However, selectivity had a slight negative 
effect on the conversion rate. 

The bottom panel in Table 4 shows the gross effects of institutional characteristics on 
personal development gains. As one might expect, institutional environments that emphasized 
personal relations had significant gross positive effects on personal development gains, as did 
emphasizing vocational and practical matters in courses. 

Science and Technology as Dependent Variable . At the institutional level, public 
institutions had slight net advantage over private institutions on student gains in science and 
technology. DUs and SLAs had smaller net effects on science and technology gains compared 
with CCUs, after controlling for EFFORTSUM. Institutions emphasizing scholarly and 
intellectual activities had positive net effects on science and technology gains and environments 
emphasizing personal relationships had a slight negative net effect. Institutions that emphasized 




scholarly and intellectual activities also had a slightly higher conversion rate from EFFORTSUM 
to science and technology gains. Emphasizing scholarly and intellectual activities had significant 
gross positive effects on science and technology gains (Table 4, bottom panel). Also, DUs and 
SLAs had smaller gross effects on science and technology gain in comparison to CCUs. 

Vocational Preparation as Dependent Variable . At the institutional level, institutions that 
emphasized vocational and practical matters had positive net effects on vocational preparation 
gains and a slightly larger slope of EFFORTSUM. However, institutions with high quality 
personal relations had a slight negative net effect on vocational preparation. Emphasizing 
vocational and practical matters also had significant gross positive effects on vocational 
preparation gains. Public institutions had a slightly smaller gross effect on vocational 
preparation than private institutions. 

Intellectual Development as Dependent Variable . Holding EFFORTSUM constant, 
selectivity had a negative effect on intellectual development gains. SLAs had a smaller net effect 
on intellectual development gains compared with CCUs. Institutions that emphasized scholarly 
and intellectual activities had positive net effects on intellectual development gains, while 
environments that emphasized personal relationships had a slight negative net effect. Institutional 
characteristics had no effect on the EFFORTSUM to GAINSUM conversion rate. While 
institutional selectivity was negatively related to the gross effects of institutional characteristics 
on intellectual development gains, emphasizing scholarly and intellectual activities had 
significant gross positive effects. 

Institutional type and institutional environment are usually correlated to some extent 
(Table 2). Therefore, in order to help understand the relationships between institutional type or 
environment and learning productivity, we reported the results from the institutional type and 
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institutional environment models in Appendix B and C. However, since the focus of this study is 
to examine the effects of institutional characteristics on student learning productivity, we did not 
report the results in Appendix B and C in detail. 

Limitations 

This study is limited in that the data are from those institutions that voluntarily 
administered the CSEQ. If data from other institutions were included the findings might change 
in unknown ways. The sampling and administration procedures likely vary across institutions, 
and are other potential sources of bias with unknown effects. It’s also possible that the results 
would change if additional student-level measures (e.g., ability, motivation) and institution-level 
data (e.g., resources) were included in the model. That said, the CSEQ research program 
represents one of the most extensive national databases with survey information from college 
students related to their quality of effort and gains from college. It is one of the few available 
sources of information from multiple institutions about the undergraduate experience that can be 
used to examine the influence of institutional characteristics on learning productivity. 

Discussion 

With these limitations in mind, three major findings from this study stand out. First, the 
largest portion of variance in the EFFORTSUM and GAINSUM learning productivity indicators 
was related to individual level variables; yet individual student characteristics explained only a 
small portion of the total variance in EFFORTSUM. Perhaps this was partly due to the absence 
of measures of individual student motivation or other important individual variables in the 
model. However, this also suggested what the students do in college has very little to do with 
what the students are in the conventional measures. That suggested the students can make their 
own efforts to make most out of colleges. 
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Second, only a small (though significant) portion of the variance in EFFORTSUM (8%) 
and GAINSUM (7%) could be attributed to institutional characteristics. However, institutional 
type and institutional environment measures accounted for a large portion of variance in 
EFFORTSUM between institutions, 39% and 44% respectively when they were included in 
modeling separately. Though the institutional environment explained a substantial amount of 
variance in GAINSUM between institutions (30.7%), it accounted for only a trivial portion of 
variance in the slope of EFFORTSUM to GAINSUM. Consistent with previous research, 
institutional type as defined by the 1 994 Carnegie scheme did not effectively account for much 
variance on GAINSUM or the slope (conversion rate) of EFFORTSUM to GAINSUM. 

Finally, the examination of the gross effects and the net effects of institutional 
characteristics (sector, selectivity, type, environment) on EFFORTSUM and GAINSUM 
suggests that some institutional characteristics may have a net positive effect on EFFORTSUM 
but a net negative effect on GAINSUM, which may lead to no gross effect on GAINSUM. The 
converse also seems to be possible in that institutional characteristics have a net negative effect 
on EFFORTSUM but a net positive effect on GAINSUM. For example, an institutional 
environment emphasizing personal relations positively affects EFFORTSUM at the institutional 
level (which in turn has a significant positive effect on GAINSUM), but it had a negative net 
effect on GAINSUM and, therefore, no gross effect on institutional average GAINSUM. 
However, an environment emphasizing vocational and practical matters had a negative effect on 
EFFORTSUM but a positive net effect on GAINSUM with no gross effect on GAINSUM. The 
only institutional characteristic that had a positive net effect on EFFORTSUM, a net positive 
effect on GAINSUM, and a gross positive effect on GAINSUM was an institutional environment 
that emphasized scholarly and intellectual activities, as indicated in the institutional environment 
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model. 



Because institutions have different missions and educational goals we can expect that 
these different foci and curricular emphases will have differential effects on student gains (Pace, 
1990b). This is evident in some of the findings from this study where institutional environments 
that emphasized scholarly and intellectual activities had gross positive effects on GAINSUM, 
general education, science and technology, and intellectual development, and where 
environments that emphasized personal relations had gross positive effects on EFFORTSUM and 
personal development. Interestingly, although institutional environment emphasizing vocational 
and practical matters had negative effects on EFFORTSUM, such environments had positive 
gross effects on personal development and vocational preparation. This seems to support 
Freeland’s (1999) endorsement of a “practice-oriented” approach to undergraduate education that 
will help students cultivate the skills needed to succeed vocational and socially after college. 

Some of these findings are seemingly at odds with what we reported previously (Kuh & 
Hu, in press b) when the student was the unit of analysis and student perceptions of institutional 
environments were treated as independent variables. The findings from our earlier study showed 
that individual student perceptions positively affected effort and gains at all institutions. More 
specifically, perceptions of all three dimensions of institutional environments had positive net 
effects on GAINSUM and all five gain factors and environments emphasizing scholarly and 
intellectual activities and vocational and practical matters had positive effects on EFFORTSUM. 
The seemingly contradictory results from the two studies may be a function of within- and 
between-institution effects, especially when most of the variance is a function of individual 
students and only a small amount of the variance can be attributed to institutional characteristics. 
Ignoring the multilevel character of the data may produce what appear to be different results. For 
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example, the between-institution comparisons in the current multilevel study showed that 
aggregated perceptions of institutional environments do have different effects on institution-level 
student average efforts and average gains. Even though institutions that emphasize vocational 
and practical matters may positively affect individual student engagement in educational 
purposeful activities, institutions that place a great emphasis on vocational and practical matters 
may still have lower institution-wide average levels of engagement. At the same time, while 
environments emphasizing vocational and practical matters may have a lower level of institution- 
wide average engagement, institutions that emphasize vocational and practical matters may 
benefit students in terms of vocational preparation and personal development. 

Pascarella and Terenzini (1991) explained that the results from different methods often 
have different implications. The findings from this study are a good example of their observation 
and underscore the need to use multi-level model when analyzing data from multiple institutions. 
Without considering the nature of nested data structure from multiple institutions (where students 
are nested within institutions), the conclusions from the conventional regression analysis could 
be misleading. Hence, analytical models such as HLM have more direct links to the conversation 
about institutional quality in promoting student learning than the conventional regression 
analysis. 

Conclusions 

The findings from this study point to three conclusions. First, the learning productivity 
model described here clearly distinguished among three paths of institutional influence on 
student learning productivity: (a) what students do in college reflected by the institutional net 
effect on EFFORTSUM, (b) institution-wide average student gain represented by the institutional 
net effect on GAINSUM, and (c) the degree to which student effort is converted into gains, 



which is reflected by the institutional net effect on the EFFORTSUM-GAINSUM conversion 
rate. This conclusion has implications for the national conversation about collegiate quality in 
that efforts to improve the quality of undergraduate education should consider ways to promote 
student learning through all three routes. Schools that leverage their resources and those of their 
students in this regard would improve their learning productivity, thus benefiting students in 
terms of greater levels of learning and personal development. Colleges and universities can 
enhance student learning by discovering more about how each contributes to desired outcomes 
and focusing institutional effort on improving the impact of each on gains. 

Second, when different gain outcomes were considered, institutional characteristics have 
varied effects on institutional level average gains and the conversion rates from effort to gains. 
That is, certain institutional characteristics may well contribute to certain types of student 
learning but become constraints on students in other types of learning. For instance, although 
institutional emphasis on vocational and practical matters can not help involve students in 
college activities, it has positive gross effect on student vocational preparations. Institutions often 
have different missions and educational goals; therefore, the conversations about institutional 
quality need to consider the mission differentiation of American colleges and universities. 

Finally, institutional environments have a greater effect on learning productivity than the 
other institutional characteristics examined in this study. This means that we need to learn more 
about role of institutional environments in fostering student quality of effort in college activities, 
in enhancing the average gains from college and improving the conversion rate of effort to gains, 
which is key to learning efficiency. We also need more sensitive and sophisticated measures of 
various aspects of institutional environments that appear to have differential effects on student 
outcomes (Astin, 1993b). The fourth edition of the CSEQ includes some additional measures 
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about institutional diversity and computing and information technology that may affect the 
relationship of effort to gains and the conversion rate. Institutions should also examine the ways 
in which various dimensions of their environments promote or hinder student learning and take 
steps to develop these positive learning-centered attributes and minimize those dimensions that 
inhibit student learning. For example, does communicating high expectations by faculty 
members and others have an effect on student perceptions of a scholarly and critical 
environment? Or is the peer culture and the perceptual environment created by being in the 
presence of other highly motivated, high ability students a factor? Even though selectivity was 
controlled, perhaps other factors are at work that affect these views. 

Implications For Research 

We agree with Johnstone: “we need to focus more on the student and his or her learning, 
and to be a little less preoccupied with, and critical of, the faculty (and all of the rest of the 
administrative, professional, and clerical support staff of our colleges and universities) in our 
quest for more productivity” (1993, p. 4). In addition to the persistent search for more resources 
and increasing selectivity, we need to learn more about how these factors and others influence 
the nature and amount of effort students devote to educational purposeful activities and how 
students use school resources into their learning. Additional institutional characteristics should 
be taken into account in such models, such as size, allocations of resources (e.g., undergraduate 
instruction, graduate student support, student support services, and overall educational 
expenditures) to determine the institutional effects on student learning productivity. The results 
of this study clearly show that it is the institutional environment, not the type of institution under 
the current Carnegie classification, that is most important in determining how much students 
learn. 




Because the gross Carnegie Foundation institutional categories fail to illuminate some 
important aspects of the undergraduate experience, this study may also have implications for 
more effectively classifying institutions that include undergraduate education in their mission. 
One approach is to determine how well schools are functioning in each of the three dimensions 
of learning productivity. Some schools may do better than others and such patterns may be 
distinctive enough to warrant creating an alternative institutional typology. A new classification 
schema based on learning productivity indicators would be an improvement over the 1 994 
Carnegie Classification of Institutions of Higher Education. Ironically, this schema has had the 
opposite effect of what was originally intended, which was to dampen trends toward 
homogenization and preserve or possible increase institutional diversity (Carnegie, 2000). 
Instead, it has had an almost pernicious effect by inducing schools to emulate those perceived to 
be more prestigious, such as the research university, a phenomenon Jencks and Riesman labeled 
“a snake-life procession.” This mission creep has had deleterious effects on goal setting, reward 
structures, and many other operational aspects of institutions of higher education that directly 
affect the amount of institutional effort directed toward undergraduate education. A logical 
extension of the learning productivity model could be an institutional typology that effectively 
differentiates colleges and universities by taking into account these three strands of institutional 
contributions to different types of student learning. Additional efforts are need to discover if 
other institutional dimensions not accounted for in the Carnegie Classification more effectively 
differentiate institutions in terms of student learning. 
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TABLE 1 . 

Sample Means, Standard Deviations, and Variable Description on Original Variables 
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Results from HLM Models: The Full Model 
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FIGURE 1. 

The Learning Productivity Model 
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Variable measured at institutional level 



APPENDIX A 

CSEQ (3rd Ed.) Scales and Items 



ACTIVITIES SCALES 

• Library (10 items) 

• Experience with Faculty (10 items) 

• Course Learning (10 items) 

• Art, Music, Theater (12 items) 

• Student Union (10 items) 

• Athletic and Recreation (10 items) 

• Clubs and Organizations (10 items) 

• Experience in Writing (10 items) 

• Personal Experiences (8 items) 

• Student Acquaintances (10 items) 

• Science (10 items) 

• Campus Residence (10 items) 

• Topics of Conversation (10 items) 

• Information in Conversations (6 items) 

ENVIRONMENT ITEMS 

1. Scholarly and Intellectual 

• Academic, scholarly, and intellectual qualities 

• Aesthetic, expressive, and creative qualities 

• Critical, evaluative, and analytical qualities 

2. Personal Relations 

• Relations among students and student groups 

• Relations with faculty members 

• Relations with administrative personnel and offices 

3. Practical and Vocational 

• Vocational and occupational competence 

• Personal relevance and practical value of courses 

GAINS ITEMS 

1. General Education 

• Understanding and enjoyment of art, music, drama 

• Acquaintance with and enjoyment of literature 

• Knowledge of history 

• Knowledge about different parts of the world and people 

• Awareness of different philosophies, cultures, ways of life 

• Broad general education 



2. Personal Development 

• Values and ethical standards 

• Self-understanding 

• Ability to get along with others 

• Teamwork skills 

• Good health habits and physical fitness 

3. Science and Technology 

• Science and experimentation 

• Science and technology developments 

• Consequences of science and technology 

4. Vocational Preparation 

• Job or work skills 

• Background for further education 

• Career information 

5. Intellectual Development 

• Writing 

• Computers and other information technologies 

• Analytical and logical thinking 

• Quantitative problem solving 

• Synthesis ability 

• Self-directed learning 



46 

o 

ERLC 



39 



Appendix B. 

Results from HLM Models: Institutional Type Model 
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Appendix C. 

Results from HLM Models: Institutional Environment Model 
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